Comparing Fifty Natural Languages and Twelve Genetic Languages Using Word Embedding Language Divergence (WELD) as a Quantitative Measure of Language Distance

摘要

We introduce a new measure of distance between languages based on wordembedding, called word embedding language divergence (WELD). WELD is defined asdivergence between unified similarity distribution of words between languages.Using such a measure, we perform language comparison for fifty naturallanguages and twelve genetic languages. Our natural language dataset is acollection of sentence-aligned parallel corpora from bible translations forfifty languages spanning a variety of language families. Although we useparallel corpora, which guarantees having the same content in all languages,interestingly in many cases languages within the same family cluster together.In addition to natural languages, we perform language comparison for the codingregions in the genomes of 12 different organisms (4 plants, 6 animals, and twohuman subjects). Our result confirms a significant high-level difference in thegenetic language model of humans/animals versus plants. The proposed method isa step toward defining a quantitative measure of similarity between languages,with applications in languages classification, genre identification, dialectidentification, and evaluation of translations.